Yes, am adding another quirk.

I've pushed a webdriver version (ignoring the 2014 quirk), but I should note that I tried it at first with PhantomJS and seemed to land on a bug (which I'm not sure how to even characterise and report), which is not a great sign.

The webdriver version is much slower. Getting the index plus 10 years takes:

• 3m56s with webdriver
• 43s with bs4

(But both could benefit from parallelism)

This suggests not selenium.

Does it have to be Python?
Another option might be PhantomJS + CasperJS (Casper can use CSS selectors or XPath)

I don't require it to be Python (though this is by far my expertise), but in the context of *Carpentry, requiring javascript knowledge rather than Python (beyond DOM) is a bit strange.

I don't expect Casper + Phantom will be much faster than using webdriver, but I may be wrong. I also don't see why you can't just use Phantom without Casper. But my experiments so far in that direction have not been a success. And as I said above, I landed on a bug on Phantom (or perhaps in webdriver.PhantomJS()) and ran away.

Yes, PhantomJS + CasperJS appears to be much faster than PhantomJS + webdriver + python. But I don't think we can afford to teach JavaScript in this context.

I'm not strongly arguing for this, but I don't think we should feel limited in our choice of languages in general terms - as long is this isn't going to create problems in terms of setup and ability for users to go away with something they can use.

There is obviously a question of any pre-requisites for people attending the course - and if we feel that experience with a specific language is a pre-requisite to doing this web scraping course it would definitely make sense for that to be Python as we already have that course in the carpentries.

And I should say I'm really appreciating your feedback @ostephens.

Should I presume that those suggesting UNSC resolutions as a target knew, or did not know, of its quirks?

Added another quirk for 2017

Thanks @jnothman :)

I don't really know if those suggesting UNSC res. as a target knew the quirks - but they definitely felt it was representative of the type of data they needed to scrape as librarians - so perhaps quirks and all is good?

That said, I don't think you should necessarily feel bound by those decisions - if no-one is here defending them we've got to still feel able to make progress :)

Okay. I'm thinking lxml is the best pick. It has some minor usability flaws relative to bs4, but:

• You need to use lxml underneath to get bs4 to parse 2014.shtml (also for scrapy), so we're not going to avoid any installation difficulties. I could be persuaded to stay with bs4 for installation reasons.
• It supports both xpath and css selectors so can be used to illustrate both if need be

And I'm wishing webdriver wasn't so sluggish as it gets around lots of pitfalls and quirks.

I've just been having a dig around and I'm inclined to agree. Potentially a mix of lxml and bs4 would be an option, but then I guess we end up trying to cover two approaches in the lesson which seems like a recipe for confusion.

Of course we could use Ruby Mechanize and Nokogiri instead :) (only joking!)

@runderwood @vphill, were the quirks of scraping the UNSC resolutions pages something you had taken into account when looking at them for use with the web scraping lesson?